Thermal imaging system with multiple selectable viewing angles and fields of view for vehicle applications

ABSTRACT

Systems and methods to improve safety and more efficient Advanced Driver Assistance Systems (ADAS) and autonomous vehicle (AV) systems for vehicular operation through the application of multiple thermal sensors arranged in systems where the resolution, Field Of View (FOV), and aiming angle of individual sensors are varied. In particular two configurations are discussed in detail, a three-sensor arrangement for forward and forward off angle data acquisition, and a two or three sensor arrangement for blind spot and pinch point awareness for towing applications. In some forward-looking three-sensor embodiments, the center sensor may provide a high-quality narrow field of view of radiometric data for use with tracking algorithms to identify pedestrian and large animal targets for long range driver identification. The side sensors may provide radiometric data for peripheral vision on short/medium range approaching targets for situational awareness at crosswalks and turning corners.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/905,172, filed Sep. 24, 2019, entitled THERMAL IMAGING SYSTEMWITH MULTIPLE SELECTABLE VIEWING ANGLES AND FIELDS OF VIEW FOR VEHICLEAPPLICATIONS, which is incorporated herein by reference in its entirety.

FIELD

The present disclosure generally relates to imaging systems includingthermal imaging sensors, and in particular to the application of thermalsensors to vehicle operation.

BACKGROUND

The increasing availability of high-performance, low-cost uncooledthermal imaging devices, such as those based on bolometer focal planearrays (FPAs), is enabling the design and production ofconsumer-oriented thermal imaging cameras and sensors capable of qualitythermal imaging. Such thermal imaging systems have long been expensiveand difficult to produce, thus limiting the employment ofhigh-performance, long-wave imaging to high-value instruments, such asaerospace, military, large-scale commercial, and automotiveapplications. Thermal imaging systems of a given design produced inquantity may have different design requirements than complex military orlarge industrial systems.

SUMMARY

The systems and methods of this disclosure each have several innovativeaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope as expressed by the claims thatfollow, its more prominent features will now be discussed briefly.

Systems and methods may be provided that improve safety and moreefficient Advanced Driver Assistance Systems (ADAS) and autonomousvehicle (AV) systems for vehicular operation through the application ofmultiple thermal sensors arranged in systems where the resolution, FieldOf View (FOV) and aiming angle of individual sensors are varied. Inparticular two configurations are discussed in detail, a three-sensorarrangement for forward and forward off angle data acquisition, and atwo or three sensor arrangement for blind spot and pinch point awarenessfor towing applications. In some forward-looking three-sensorembodiments, the center sensor may provide a high-quality narrow fieldof view of radiometric data for use with tracking algorithms to identifypedestrian and large animal targets for long-range driveridentification. The side sensors may provide radiometric data forperipheral vision on short/medium range approaching targets forsituational awareness at crosswalks and when approaching or turningcorners.

In one aspect, a system for enhancing vehicular operation may beprovided, including: a sensor carrier configured for mounting to atleast one selected location on a vehicle; at least two thermal imagingsensors, each sensor comprising an thermal imaging Focal Plane Array(FPA) and associated interface and signal processing elements, each FPAincluding a number of pixels defining the image resolution of the FPA,configured for mounting to the sensor carrier; optics associated witheach sensor defining the Field of View (FOV) of each sensor; wherein,one sensor with a first number of pixels and a first defined FOV isdisposed in the carrier at a first viewing angle, and a second sensorwith at least one of the same or a different number of pixels and asecond defined FOV disposed in the carrier at a second viewing angle.

In one embodiment of the first aspect, the sensor configuration mayinclude one or more of; a camera core with shutter and optics; ashutterless camera core; a microchip mounted to a substrate. In anotherembodiment of the first aspect, the carrier configuration may includeone or more of; a separate package configured to mount at least one ofcores, optical elements, or substrates; mounting provision in anothervehicle element. In one embodiment of the first aspect, the system mayfurther include three sensors, wherein one sensor may be configured tobe higher resolution compared to the other two sensors, with a narrowerFOV than the other two sensors, disposed with a viewing axis alignedsubstantially to the vehicle's forward orientation, and the other twosensors may be lower resolution, with a wider FOV disposed with viewingaxes at an angle to both sides of the vehicles forward orientation.

In another embodiment of the first aspect, the forward looking sensormay be configured to have a FOV angle of less than 30 degrees, including24 degrees and the angled sensors have a viewing axis of greater than 20degrees, including the range of 28 to 30 degrees from the forwardlooking sensor viewing axis, and a FOV of greater than 45 degrees,including 55 degrees. In one embodiment of the first aspect, the forwardlooking sensor may be configured to have a FOV angle of substantially 24degrees and the angled sensors may have a viewing axis of substantiallywithin the range of 28 to 30 degrees from the forward looking sensorviewing axis, and a FOV of substantially 55 degrees.

In another embodiment of the first aspect, the forward-looking sensormay be configured to view a range of less than 600 feet, and the sidelooking sensors are configured to view a range of less than 200 feet. Inone embodiment of the first aspect, the forward-looking sensor may beconfigured to view a range substantially within 378 to 573 feet, and theside looking sensors may be configured to view a range substantiallywithin 101 to 161 feet. In another embodiment of the first aspect, theforward-looking sensor may include a QVGA FPA and the two side lookingsensors may include a 200×150 pixel FPA.

In one embodiment of the first aspect, image data acquired by thesensors may be provided to a vehicle control/display system and acquiredsensor data may be at least one of; displayed simultaneously to adriver; selectively displayed to a driver; or processed to provide atleast one of driver warnings or assisted driving actions. In anotherembodiment of the first aspect, sensor viewing angle may be adjusted inboth orientation relative to forward vehicle axis and relative to theplane of the road to account for mounting position on the vehicle anddesired FOV. In one embodiment of the first aspect, the adjustments maybe at least one of fixed at installation or dynamically adjustableduring use.

In another embodiment of the first aspect, the system may be configuredas at least one of; at least one sensor directly forward looking and atleast one sensor looking off angle to the forward looking sensor,mounted to view forward; at least one sensor looking backward and atleast one sensor looking off angle to the rear viewing sensor, mountedto at least one of the rear or to one side of a vehicle; or a pluralityof sensors mounted to look at 360 degrees around the vehicle, andmounted and aimed to cover desired FOV's. In one embodiment of the firstaspect, the sensor configuration may be three shutterless thermal cameracores mounted in a package whose footprint is less than 20×40 mm. Inanother embodiment of the first aspect, the sensor configuration may bethree shutterless thermal camera cores mounted in a package whosefootprint is substantially 16×36 mm.

In one embodiment of the first aspect, the system may be powered by andinterfaced to the vehicle's control and display system, and is designedinto the vehicle. In another embodiment of the first aspect, the systemmay be an aftermarket accessory and is at least one of configured with adedicated processor and display or interfaced to the vehiclecontrol/display system. In one embodiment of the first aspect, thesensors may be battery powered configured for the placement of thesensors in a temporary location depending on need and the system iswirelessly interfaced to at least one of a dedicated controller or thevehicle controller.

In another embodiment of the first aspect, the system may include amount on the forward end of the roof configured to minimize thecollection of debris as housing placement on the front of the roofprovides access to the windshield fluid system for cleaning. In oneembodiment of the first aspect, the optics may include at least one lensas an exterior element, comprising a diamond-like coating (DLC) on atleast the exterior facing portion of the lens.

In a second aspect a method may be provided for using sensor systems asdescribed above to enhance vehicle operation, given the system may beinterfaced to a vehicle control and display system, including: producingat least one of thermal images or thermal image derived data of space infront of a vehicle and of the space to the forward right and left of thevehicle; and providing at least one of thermal data display, warnings,or driver assist operations if objects are detected in the direction oftravel or in the direction of an indicated turn.

In a third aspect method may be provided of using any combination of thesensor systems as described above to enhance operation of vehiclesengaged in towing, given the system may be interfaced to a vehiclecontrol and display system, including: producing at least one of thermalimages or thermal image derived data of space directly to the rear rightof a vehicle using a lower FOV sensor and of the space to the rearwardright of the vehicle using an angled higher FOV sensor; and providing atleast one of thermal data display, warnings, or driver assist operationsif objects are detected in to the rear or in the direction of anindicated turn.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned aspects, as well as other features, aspects, andadvantages of the present technology will now be described in connectionwith various implementations, with reference to the accompanyingdrawings. The illustrated implementations are merely examples and arenot intended to be limiting. Throughout the drawings, similar symbolstypically identify similar components, unless context dictatesotherwise.

FIG. 1 illustrates a general arrangement of an example multisensor/multi FOV imaging system.

FIG. 2 illustrates details of an example multi sensor/multi FOV imagingsystem.

FIG. 3 illustrates the operation of an example multi sensor/multi FOVimaging system.

FIGS. 4A and 4B illustrate the application of a forward-looking vehicleapplication of an example multi sensor/multi FOV imaging system.

FIGS. 5A and 5B show hazardous aspects of vehicles engaged in towingoperations.

FIGS. 6A and 6B illustrate the application of a rear/side looking towingapplication of an example multi sensor/multi FOV imaging system.

FIGS. 7A, 7B, 7C, and 7D show various exemplary embodiments multisensor/multi FOV imaging systems.

FIG. 8 illustrates details of a particular embodiment of a multisensor/multi FOV imaging system.

FIGS. 9A and 9B illustrate potential adjustment features for an examplemulti sensor/multi FOV imaging system.

FIGS. 10A, 10B, and 10C show various exemplary embodiments for vehicleintegration for multi sensor/multi FOV imaging systems.

FIG. 11 illustrates another example embodiment of a three-sensorforward-looking multi senor/multi FOV imaging system.

FIG. 12 illustrates another view of the embodiment of FIG. 11.

FIG. 13 illustrates an example of performance trade-offs associated withsensor resolution.

FIG. 14 is a block diagram illustrating an example multi-sensor system.

DETAILED DESCRIPTION

The following description is directed to certain implementations for thepurpose of describing the innovative aspects of this disclosure.However, a person having ordinary skill in the art will readilyrecognize that the teachings herein can be applied in a multitude ofdifferent ways.

Generally described, embodiments of the present disclosure relate toapplying thermal imaging to vehicles. Thermal imaging has the potentialto provide significant safety advantages for moving vehicles. Inparticular, at night or within limited visibility zones, thermal imaginghas the potential to provide discrimination of type of objects notpossible with other vehicle sensors. An example is a person who is in aposition where they could potentially be in a vehicle's path but arecurrently not illuminated by headlights, such as a child off to the sideof a road at night. Visible imagers, radar, or LIDAR may all detect thechild, but often will not be able to determine whether it is a person,or an inanimate object such as a fire hydrant. For such a situation, thevehicle operator may decide to turn just as the child starts to move andby the time the child is positively identified under headlightillumination, it may be too late to avoid contact. This is just one ofmany examples of how thermal imaging may provide additional crucialinformation to that currently available to drivers.

In the past, the expense and complexity of interpreting thermal datahave limited the adoption of mass market thermal imagers for vehicles.Recent advances in thermal imaging technology have greatly reduced thecost, size, and environmental susceptibility of thermal imaging sensorswhile the ability to extract and present usable information from thermalimages has increased. It is now reasonable to contemplate vehiclesolutions involving thermal imaging that are suitable for automotive usein terms of ruggedness and environmental resistance. With the decreasedcost and size, it is also possible to contemplate thermal imagingsolutions that may entail a plurality of sensors which can be configuredspecifically to real world vehicle situations. For applying thermalimaging sensors to vehicle operations, it may be desirable to mixmultiple sensors of differing characteristics to provide more completethermal information about the space a vehicle is traveling through.

In general, the present disclosure addresses multiple integrated thermalsensors encompassing simultaneous fields of view, and in particularembodiments, including wide horizontal peripheral vision and narrowdown-the-road vision for a high-performing ADAS, Automatic EmergencyBraking (AEB), and/or autonomous vehicle (AV) cost-effective ExteriorFar Infrared Imaging system.

FIG. 1 shows a general depiction of some example embodiments presentedin this disclosure. A thermal imaging system 100 suitable for forwardlooking in a vehicle is shown. There are three areas that areparticularly relevant. The forward FOV 115 is in the direction thevehicle is currently pointed. Relevant information to the driver in thisFOV should preferably be good out to a long range, say several hundredfeet (e.g., all or part of the usable headlight range, which may be asmuch as 1000 feet out). It should preferably be high resolution toprovide early discrimination at high speeds. Thus, a narrow FOV, withpotentially a high-resolution imager, may be most suitable for thedirect ahead view.

Also shown in FIG. 1 are two side FOV's 125. The side FOV's 125represent a space that the vehicle could possibly occupy in the nearfuture, such as in making a left or right turn. The requirements forthese FOV's may be different than for the forward FOV 115. It is lessimportant to see a long distance at high resolution and more importantto view a wide area with enough resolution to identify the heatsignature of a living body within the rather large area that a turningcar can potentially occupy during a turn or the space to the side of thevehicles forward motion where living bodies can step out of the darknesswithout warning. Thus, the side FOV's should preferably be wide, can beshorter, and since objects of interest in these FOV's may often becloser to the vehicle, high resolution may be less important.

With these issues in mind, it is possible that the solution for forwardlooking thermal imaging driver assistance may not be adequately providedby a single imager with one set of range/FOV defining optics.

FIG. 2 shows an approach to a thermal imaging solution to theforward-looking driving scenario. In FIG. 2, there are three imagingsensors which may be of two types 110 and 120. Three sensors is a usefulconfiguration for forward-looking applications, for example, but othernumbers of sensors and/or sensor types are possible. It is envisionedthat these sensors are based on microbolometer based Focal Plane Arrayswhich can be produced cheaply and economically, and have suitablecharacteristics for long wave thermal imaging. Such FPA's are sold inquantity by the owner of the current disclosure and are discussed indetail in a number of patents and patent applications owned by the ownerof the current disclosure as well as numerous other patent andnon-patent publications. It is envisioned that these FPA's are packagedin a carrier, which could be a purpose-dedicated housing just for theFPA's containing the FPA's and any associated optics and interfaceelectronics for power and communication with a controller and/ordisplay. It is also possible that the carrier could be designed to mountinto an existing vehicle element such as a forward-looking visiblecamera mounting. Various carrier configurations are possible and in andof themselves are not critical to the scope of the present disclosure.

Although other integration scenarios are possible and useful, as will bedescribed later in this disclosure, for simplicity for the time being wewill discuss the vehicle thermal imaging system for the scenario wherethe thermal imaging system is part of the vehicle design, and ispermanently mounted in a suitable location, and is powered by thevehicle and is controlled by and communicates thermal image data to thevehicle control and display system. In other embodiments of the presentdisclosure, the thermal imaging system may be a modular or third-partyaddition to a vehicle, may be mounted temporarily within the vehicle,and/or may include an independent power source.

The FPA's themselves may be packaged in a variety of ways. As typicalFPA's are microchips, they could be mounted in the carrier directly ontosubstrates mounted to the carrier, and the various optics and associatedconnecting and hardware residing on the carrier. Such a solution may bethe smallest, least expensive approach. An alternative, but faster toimplement, approach that still meets the size and cost requirements isto provide the FPA's as part of a camera core. Such cores are alsodelivered in quantity and discussed in many patent and non-patentpublications by the current disclosure owners. Basically a camera coreis a package containing the various elements needed to operate the FPA'sand may include some or all of power conditioning, optical elements suchas lenses, shutters, apertures and FOV limiting bores, clock generation,and in the most useful scenarios significant local processing that inmany cases the interface from the vehicle to the core may be as simpleas power in, video out, and possibly a few parallel or serial controllines. Some automotive video interfaces include the option of embeddedcamera control within the video data communications scheme. The signalprocessing to create usable image data may be accomplished in the corein some cases. At any rate, as modern vehicles are already set up toinclude exterior camera image data to use for driver assistance and/ordisplay, integrating video streams of thermal image data is practicalfor modern vehicles.

A particularly applicable core suitable for the embodiments of thecurrent disclosure is a shutterless microcore, of a type described inPCT Application No. PCT/US2018/038606. These particular cores areapproximately cube shaped and easy to mount, inexpensive, very small,less than 0.5 “on a side, and have built in optics. So, oneimplementation of system 100 is to use three microcores for elements 110and 120, packaged in a housing or carrier with their lenses pointing inthe desired directions, and their signals rerouted to the vehiclecontroller.

A shutterless core may be less expensive, more rugged and more reliablethan a shuttered system. Moreover, shutterless cores are not subject tothe intermittent “blindness” that affects shuttered cores during theframes that the shutter is closed. This means shutterless cores have thepotential to be more reliable and safer to use in a fast-movingautomotive environment. To make up for the lack of shutter-basednon-uniformity correction (NUC), the shutterless designs may implementone or a combination of calibration based and/or scene based NUC asdescribed in the co-pending PCT Application No. PCT/US2018/038606.

Accordingly, for the case shown in FIG. 1, there may be one imagingsensor 110 for the center direction and two imaging sensors 120 for theside looking directions. The center imaging sensor FOV 115 may need tobe long range and high resolution, but not necessarily very wide. Thus,it is envisioned that the center imaging sensor 110 may possibly havehigher pixel resolution (e.g., more pixels on the FPA), but regardlessof pixel resolution, its optics may be configured for a narrower FOV 115compared to the side-aimed imagers. Side or peripheral imaging sensors120 may have relatively wider FOVs 125 and may optionally havecorresponding lower resolution. Thus, the use of multiple sensors withdifferent FOVs/resolutions allows for a tailored solution to theforward-looking vehicle thermal imaging scenario. Details of eachsensor's resolution, aiming angle, and FOV will be discussed later forsome particular embodiments, but the base concept supports a wide rangeof application dependent implementation details. Of course, it ispossible to configure FOV and resolution with optics and not necessarilyneeding to use FPA's of different pixel counts or other physicaldifferences.

FIG. 3 shows a practical application of the arrangement of FIGS. 1 and2. Center imager 110 FOV 115 is narrow and high resolution compared toside looking imagers 120 FOV's 125. Thus, more pixels are devoted to anobject (the man in the center) for the center imager 110 than for thesame object at the same distance for the side looking imagers 120. Itmay be desirable to align the FOV's so there is overlap from a suitabledistance out (the front of the vehicle or nearly so). If the images forall three were displayed stitched together at once for similarly sizedobjects at the same distance, the image would appear as shown in FIG. 3.

It is possible and maybe desirable for some cases to actually displaythe entire composite image. However, it may be more useful to presentthe data to the vehicle controller and use it intelligently. Forexample, most of the time, the side image data is not useful and mayjust add confusion. Even the center thermal image may not be useful formuch of the time. Current vehicles don't generally display their visualcamera and radar data in image form, but many display it in iconic formshowing the relationship of the surrounding space to the vehicle. Thisinformation is used for driver awareness, collision warning, and forsome vehicles, driver assist safety operation such as automatic brakingand side collision avoidance. Such use of the thermal imaging data wouldbe advantageous, and in particular for the arrangement shown the abilityto discriminate living objects in the side FOV's would be a significantsafety enhancement.

FIGS. 4A and 4B show the application of the forward-lookingmulti-sensor/multi FOV thermal imaging system to a vehicle. System 100may be integrated with vehicle 210 in a variety of ways. A simpleapproach as shown is just to add it either within the existing case ormounted on or adjacent to the forward looking visible camera housingoften found at or near the rear view mirror point, in the front grill,on the forward part of the roof, or in other convenient locations.Obviously, the system 100 is not shown to scale. The data provided bythe system 100 to the vehicle control system may desirably be compatiblewith image data used for detection, warnings and driver assistance isalready present in many modern vehicles, and not just in the moreexpensive models.

FIGS. 5A and 5B illustrate another vehicle application that couldbenefit from the multi-sensor/multi-FOV thermal imaging approach. Towingapplications, of which Semi trucks are among the most dangerous, sufferfrom several very dangerous attributes. As shown in FIG. 5A, the driverin a towing application is limited by several blind spots due to thegeometry of the vehicle and trailer. Moreover, as shown in FIG. 5B themechanics of a trailer making a 90 degree turn cause uncontrolled motionof the trailer directly into blind spot areas, as well as “pinch point”behavior of the cab follows too tight a radius. At night, pedestrians orcyclists may be invisible to a driver even when not in these “exclusionzones,” and may be virtually undetectable when in them.

FIG. 6A shows a multi-sensor/multi-FOV 100 arrangement suitable fortowed applications. In this scenario, two side-looking wide FOV sensorsare arranged to view the area in the blind spot and turning radiusdanger zones. Optionally a narrow FOV sensor could be employed to lookstraight back along the trailer. As shown in FIG. 6B the sensor system100 could be mounted on one or both sides of the vehicle 210. The sidemirror mounts would be one suitable location. Such an arrangement wouldbe capable of determining if living warm objects were in the blind spotsor pinch point zones.

FIGS. 7A, 7B, 7C, and 7D show a variety of arrangements for forwardlooking, rear looking, combined side and 360-degree coverage. Allarrangements would benefit from tailoring the resolution and FOV to theexpected scenarios at each viewing angle.

FIG. 8 shows one detailed example forward/peripheral thermal imagingsystem embodiment of the forward-looking arrangement of FIG. 2. In thisdetailed embodiment, one QVGA shutterless microcore is used for thecenter sensor and two shutterless 200×150 pixel microcores are used forthe side angle sensors. It will be understood that other types ofmicrocores (e.g., shutterless 208×156 pixel microcores) may equally beimplemented. All three with associated optics and support hardware fitinto an easily mountable package 16 mm×36 mm×8 mm in dimension. One 15pin connector is adequate for video, power and control signals. TheFOV's and orientation angles shown provide proper coverage for thescenarios described above for the forward-looking system. Obviously, theexact numbers shown are not critical, but are merely exemplary, and anyvalues with a reasonable range of those shown will achieve suitableresults. Such a device is easily mountable in many suitable locations,can be made in quantity for a reasonable cost and supplies data easilyworked with by modern vehicle controllers with driver assistcapabilities.

Depending on the application the pointing of the sensors (110, 120,e.g., cores, FPA's, optical elements, etc.) may require adjustment. FIG.9A shows imaging system 100 adjustment relative to the vehicle motionaxis, which we have already implied would be set differently for eachvehicle. However, it may also be desirable to adjust the pointing of thesensors, as shown in FIG. 9B, up or down relative to the ground. Forexample, a 360-degree view system such as shown in FIG. 7D may have tobe mounted high enough to clear all viewing constructions so it may bedesirable to aim the sensors down to compensate. Similarly, if arear-side looking system is mounted on a semi-truck mirror mount, it maybe desirable to aim each sensor at a different up/down angle dependingon the location of the exclusion zones vertically.

There are a variety of options for integrating themulti-sensor/multi-FOV imaging system 100 with a vehicle 210 as shown inFIGS. 10A, 10B, and 10C. One advantageous way is to design the systeminto the vehicle from the beginning to get the maximum benefit of havingall the angles and FOV's tailored to the vehicle and mounting location,as well as having the data available and usable to the vehiclecontroller for all the reasons discussed above. This configuration isshown in FIG. 10A, where system 100 is mounted at manufacture to thevehicle 210 and seamlessly power and signal interfaced to the vehiclecontroller 220.

However, there is no reason that the system can't be an aftermarketaccessory akin to back-up cams and dash cams which with a bit of effortcan be wired into an existing vehicle. In this arrangement shown in FIG.10B, There may be a dedicated controller/display 150 system which may ormay not be connected to vehicle controller 220. The dedicated controllercould be a user smartphone, tablet, or the like or could bepurpose-built accessory. The connection to the controller could bewireless or wired. In this scenario, that adjustability shown in FIGS.9A and 9B may be important as each vehicle/mounting position may requiredifferent aiming of the sensors.

As shown in FIG. 10B, for some applications it may be desirable to makeall or part of the sensor system 100 removable or temporarilyinstallable. For instance, towing applications may require a rearlooking sensor be placed when a trailer is hooked up. For example, maysemi drivers don't always pull the same trailer so the driver may wantto temporarily install sensors on each trailer he contracts to tow. Inthis scenario, the removable portion of the system may be batterypowered and interface wirelessly in some embodiments.

A particular example solution for a forward/peripheral lookingmulti-thermal sensor system 100 is shown in FIGS. 11 and 12. For thissolution, the sensors are shutterless micro-cores providing a horizontalcomposite 180-degree FOV using three (3) 200×150 pixel sensors, onepointing forward and two pointing peripherally. The two peripheralsensors 120 use a 1.9 mm f/1.1 lens providing a peripheral FOV 125 of81° H×59° V. each. The center sensor 110 each use a 6.5 mm f/1.0 lensproviding a center FOV 115 of 24° H×16° V. The sensors are mountedco-axially vertically (V) and with a two degree overlap horizontally (H)providing a 180-degree horizontal composite field of view (HFOV).

The system 100 for this particular embodiment may be a roof mounteddesign Roof Design, and the package for this particular configuration is28×43.6×22.5 mm L×W×H, ˜19,000 cubic mm or ˜1.16 cubic inch, in atrident shape as shown. A mount on the forward end of the roof mayminimize the collection of debris as housing placement on the front ofthe roof provides potential access to the windshield fluid system forcleaning.

Also envisioned for the exemplary embodiment is to utilize adiamond-like coating (DLC) on the optics instead of a more typicaloutside window. This arrangement removes a temperature delta between theoptics and window (typically germanium) allowing for higher performance,including better Modulation Transfer Function (MTF) for improved objectdetection. This coating allows for the optics to be implemented as atleast one exterior facing lens, which in some cases may be the onlyoptics needed. The lens coating will enable a single exterior facinglens, coated on at least the exterior facing side of the lens, as thecoating will protect against impacts from road debris and will allow forconvenient cleaning due to inherent abrasion resistance.

It is envisioned that system 100 will include two or more thermalimaging sensors interfaced to a controller that at a minimum performscontrol, configuration and calibration data management of the thermalsensors, resulting in either three serial video streams or possibly oneserial video stream comprising a stitched together video stream of thecomposite image from the imaging sensors. These control functions couldbe performed in any suitable logic or programmable device including anFPGA, ASIC, or processor.

A Block Diagram showing generally the elements likely to be needed isshown in FIG. 14, for a three sensor 110/120 system 100. The sensors110/120 can be controlled and accessed by one or more logic elements1020 such as programmable controllers, or for some cases FPGA's orASIC's. A Memory element such as flash memory 1010 can be provided forstoring programming steps, calibration data or other necessary dataneeded for operation. Finally, the control logic can format the videodata from the sensors as appropriate and via a transceiver 1030, forexample a serial video data transceiver, and can pass the data to thevehicle controller 220. One common data link for automotive ADAS/videocamera interfaces is FPD-Link, while another common camera interface isGMSL (gigabit multimedia serial link). Other auto manufacturers may usecommon data interfaces such as USB and ethernet. Depending on thepreference of the vehicle designer, the system may pass any type ofprocessed video to the vehicle controller ranging from raw video tofully imaged processed video with some or all of the common thermalimage processing elements such as Non-Uniformity Correction (NUC). noisefiltering, thermography and the like. The signal processing trade-offbetween the vehicle controller and the imaging system itself can drivethe system design, where for less processing at the system level,simpler logic elements such as FPGA's may suffice, while if fullyprocessed video is delivered to the vehicle, more powerful andprogrammable logic may be needed.

Typical automotive video connectivity can include 8-bit data streamscombined into a single serializer used for visible cameras (e.g., RGB).For the thermal system 100 embodiment, three separate data outputs(e.g., R=left side sensor, G=center sensor and B=right side sensor) maybe combined onto a single serializer for multiple sensor output, as oneparticularly convenient approach to passing multi camera dataefficiently.

It is also possible as described above, to utilize differing pixelresolution sensors for the center and peripheral imagers in theembodiment of FIGS. 11 and 12. FIG. 13 shows an example of resolution atrange improvement 115 a compared to 115 b, when, for example, the centersensor is replaced with a QVGA thermal camera as opposed to the 200×150sensor described above. Any suitable sensor, including higher pixelresolution imagers such as HVGA, VGA, or higher may be suitable for anyof the sensors in any of the embodiments disclosed herein, depending onthe intended use if the system.

Depending on the embodiment, certain acts, events, or functions of anyof the processes described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of thealgorithm). Moreover, in certain embodiments, acts or events can beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, and process stepsdescribed in connection with the embodiments disclosed herein can beimplemented as electronic hardware, computer software, or combinationsof both. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. The described functionality can be implemented invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the disclosure.

The various illustrative logical blocks and modules described inconnection with the embodiments disclosed herein can be implemented orperformed by a machine, such as a processor configured with specificinstructions, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A processor can be amicroprocessor, but in the alternative, the processor can be acontroller, microcontroller, or state machine, combinations of the same,or the like. A processor can also be implemented as a combination ofcomputing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration. For example, the LUTdescribed herein may be implemented using a discrete memory chip, aportion of memory in a microprocessor, flash, EPROM, or other types ofmemory.

The elements of a method, process, or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of computer-readablestorage medium known in the art. An exemplary storage medium can becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium can be integral to the processor. The processor andthe storage medium can reside in an ASIC. A software module can comprisecomputer-executable instructions which cause a hardware processor toexecute the computer-executable instructions.

Conditional language used herein, such as, among others, “can,” “might,”“may,” “e.g.,” and the like, unless specifically stated otherwise, orotherwise understood within the context as used, is generally intendedto convey that certain embodiments include, while other embodiments donot include, certain features, elements, and/or states. Thus, suchconditional language is not generally intended to imply that features,elements and/or states are in any way required for one or moreembodiments or that one or more embodiments necessarily include logicfor deciding, with or without author input or prompting, whether thesefeatures, elements and/or states are included or are to be performed inany particular embodiment. The terms “comprising,” “including,”“having,” “involving,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Disjunctive language such as the phrase “at least one of X, Y, and Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y or Z, or any combination thereof (e.g., X, Y and/or Z).Thus, such disjunctive language is not generally intended to, and shouldnot, imply that certain embodiments require at least one of X, at leastone of Y or at least one of Z to each be present.

Unless otherwise explicitly stated, articles such as “a” or “an” shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

While the above detailed description has shown, described, and pointedout novel features as applied to illustrative embodiments, it will beunderstood that various omissions, substitutions, and changes in theform and details of the devices or processes illustrated can be madewithout departing from the spirit of the disclosure. As will berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. A system for enhancing vehicular operation,comprising: a sensor carrier configured for mounting to at least oneselected location on a vehicle; at least two thermal imaging sensors,each sensor comprising a thermal imaging Focal Plane Array (FPA) andassociated interface and signal processing elements, each FPA includinga number of pixels defining the image resolution of the FPA, configuredfor mounting to the sensor carrier; and optics associated with eachsensor defining the Field of View (FOV) of each sensor; wherein onesensor with a first number of pixels and a first defined FOV is disposedin the carrier at a first viewing angle, and at least one second sensorwith a at least one of the same or a different number of pixels and asecond defined FOV disposed in the carrier at a second viewing angle. 2.The system of claim 1, wherein the at least two thermal imaging sensorscomprise one or more of: a camera core with shutter and optics; ashutterless camera core; or a microchip mounted to a substrate.
 3. Thesystem of claim 2, wherein the sensor carrier includes one or more of: aseparate package configured to mount at least one of cores, opticalelements, or substrates; or mounting provision in another vehicleelement;
 4. The system of claim of claim 1, comprising three sensors,wherein one sensor is configured to be higher resolution compared to theother two sensors, with a narrower FOV than the other two sensors,disposed with a viewing axis aligned substantially to the vehicle'sforward orientation, and the other two sensors are lower resolution,with a wider FOV disposed with viewing axes at an angle to both sides ofthe vehicles forward orientation.
 5. The system of claim 4 wherein theforward looking sensor is configured to have a FOV angle of less than 30degrees, including 24 degrees and the angled sensors have a viewing axisof greater than 20 degrees, including the range of 28 to 30 degrees fromthe forward looking sensor viewing axis, and a FOV of greater than 45degrees, including one of 81 or 55 degrees.
 6. The system of claim 5wherein the forward looking sensor is configured to have a horizontalFOV angle of substantially 24 degrees and the angled sensors have aviewing axis of substantially to overlap by 28 to 30 degrees from theforward looking sensor viewing axis, and a horizontal FOV ofsubstantially 81 degrees, providing a total composite horizontal FOV(HFOV) of 180 degrees.
 7. The system of claim 5 wherein theforward-looking sensor is configured to view a range of less than 1000feet, and the side looking sensors are configured to view a range ofless than 200 feet.
 8. The system of claim 7 wherein the forward-lookingsensor is configured to view a range substantially within 378 to 573feet, and the side looking sensors are configured to view a rangesubstantially within 101 to 161 feet.
 9. The system of claim 5, whereinthe forward-looking sensor comprises at least one of a QVGA FPA or a200×150 pixel FPA and the two side looking sensors each comprise a200×150 pixel FPA.
 10. The system of claim 4, wherein image dataacquired by the sensors is provided to a vehicle control/display systemand acquired sensor data is at least one of: displayed simultaneously toa driver; selectively displayed to a driver; or processed to provide atleast one of driver warnings or assisted driving actions.
 11. The systemof claim 1 wherein sensor viewing angle may be adjusted in bothorientation relative to forward vehicle axis and relative to the planeof the road to account for mounting position on the vehicle and desiredFOV.
 12. The system of claim 9, wherein the adjustments are at least oneof fixed at installation or dynamically adjustable during use.
 13. Thesystem of claim 1, wherein the system is configured as at least one of:at least one sensor directly forward looking and at least one sensorlooking off angle to the forward-looking sensor, mounted to viewforward; at least one sensor looking backward and at least one sensorlooking off angle to the rear viewing sensor, mounted to at least one ofthe rear or to one side of a vehicle. a plurality of sensors mounted tolook at 360 degrees around the vehicle, and mounted and aimed to coverdesired FOV's.
 14. The system of claim 3 wherein the sensorconfiguration is three shutterless thermal camera cores mounted in apackage whose footprint is less than 20×40 mm.
 15. The system of claim14 wherein the sensor configuration is three shutterless thermal cameracores mounted in a package whose footprint is substantially 16×36 mm.16. The system of claim 1 wherein the system is powered by andinterfaced to the vehicle's control and display system, and is designedinto the vehicle.
 17. The system of claim 1 wherein the system is anaftermarket accessory and is at least one of configured with a dedicatedprocessor and display or interfaced to the vehicle control/displaysystem.
 18. The system of claim 1 wherein the sensors are batterypowered, configured for the placement of the sensors in a temporarylocation depending on need and the system is wirelessly interfaced to atleast one of a dedicated controller or the vehicle controller.
 19. Amethod of using a sensor system as described in claim 4 to enhancevehicle operation, wherein the system is interfaced to a vehicle controland display system, comprising: producing at least one of thermal imagesor thermal image derived data of space in front of a vehicle and of thespace to the forward right and left of the vehicle; providing at leastone of thermal data display, warnings, or driver assist operations ifobjects are detected in the direction of travel or in the direction ofan indicated turn.
 20. A method of using any combination of the sensorsystems as described in claim 13 to enhance operation of vehiclesengaged in towing, wherein the system is interfaced to a vehicle controland display system, comprising: producing at least one of thermal imagesor thermal image derived data of space directly to the rear right of avehicle using a lower FOV sensor and of the space to the rearward rightof the vehicle using an angled higher FOV sensor; providing at least oneof thermal data display, warnings, or driver assist operations ifobjects are detected to the rear or in the direction of an indicatedturn.
 21. The system of claim 5, wherein the sensor carrier isconfigured for mounting on a forward end of a roof of the vehicle tominimize the collection of debris and provide access to a windshieldfluid system for cleaning.
 22. The system of claim 5, wherein the opticscomprise at least one lens as an exterior element and a diamond-likecoating (DLC) on at least an exterior-facing portion of the lens.